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ABSTRACT 



The Surveys of Enacted Curriculum (SEC) are designed to 
provide reliable, comparable data that are collected from teachers and 
students at the K-12 math and science classroom level. The data analysis and 
reporting tools are intended to assist teachers, administrators, and 
policymakers with planning for instructional improvement in several ways: (1) 

align curriculum with standards and system-wide assessments; (2) monitor 
indicators of instruction and the relationship to student achievement; (3) 
analyze differences in instruction and content across schools and classes and 
identify improvement strategies through school leadership teams; and (4) 
evaluate the effects of initiatives such as professional development in 
changing math and science practices. (MVL) 
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Surveys of Enacted Curriculum: Tools and Services 
to Assist Educators 




Product Description 

The Council of Chief State School Officers (CCSSO) and Andrew Porter and John Smithson of Wisconsin 
Center for Education Research (WCER) have partnered to develop an advanced, in-depth approach to 
collecting and reporting data on the “enacted curriculum” in K-12 math and science, i.e. the actual subject 
content and instructional practices experienced by students in classrooms. We have conducted research 
and testing of enacted curriculum survey tools with schools and teachers. As a result, we are now able to 
offer the survey tools and a range of related data services to states and districts. 

The Surveys of Enacted Curriculum (SEC) are designed to provide reliable, comparable data that are 
collected at the classroom level with teachers and students. The data analysis and reporting tools are 
intended to assist teachers, administrators, and policymakers with planning for instructional improvement 
in several ways: 

1) Align curriculum with standards and system-wide assessments 

2) Monitor indicators of instruction and relationship to student achievement 

3) Analyze differences in instruction and content across schools and classes, and identify 
improvement strategies through school leadership teams 

4) Evaluate effects of initiatives, such as professional development, in changing math and 
science practices. 

The SEC tools and services have been developed since the mid-1990s with assistance from many 
experts and educators. The data collection instruments were field-tested in several hundred schools. 

The development process was supported by states and by the National Science Foundation. 
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Services Available from CCSSO / WCER 



Teacher and student surveys: The survey instruments are designed for self-report by teachers 
in math and science at elementary, middle or high school levels. Two major components of the teacher 
surveys are: a) Subject content of instruction, and b) Instructional activities and teacher preparation. 
Student survey items assist in validation, and are typically administered to a sub-sample of classes. A 
school system could choose sections of the teacher surveys for their specific needs. The complete 
survey requires average of 1 .5 hrs. for a teacher to complete. An on-line, web-based survey system is 
available. 



Data collection, input, and analysis: Field tests 
and prior data collection show distinct advantages of 
survey administration on-site to all teachers in a sample of 
schools. Teachers and administrators should receive 
orientation on the intended uses of data for evaluation and 
school improvement. The web-based survey will facilitate 
data entry, analysis and reports. Or, paper forms can be 
used, with data input through a scanner. Data analyses 
focus on differences in instructional practices and 
curriculum content across classrooms, schools, and 
districts. Item profiles and scales are analyzed by 
characteristics of schools (e.g., level, enrollment, reform 
initiative, student demographics) or by teacher 
characteristics (e.g., high professional development in 
subject vs. low professional development). 
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Reporting formats for various purposes. User-friendly graphics and charts have been 
developed to facilitate comparison of math and science instruction with state standards or local goals for 
improvement. School leadership teams, curriculum specialists, administrators, and policymakers can 
use the survey data reports to track effects of initiatives on classrooms, or identify areas of improvement 
efforts. The CCSSO/WCER analysis and report methods include analysis of alignment of instruction with 
state or local assessment instruments. 



Designs for professional development and 
assistance to schools. CCSSO/WCER provides models 
for workshops with teachers and school leaders based on 
use of enacted curriculum data and reports. Our designs 
include strategies for leadership in analysis of data within 
schools and methods of using cooperative inquiry-based 
approaches for improving instruction (see “Using Data on 
Enacted Curriculum-A Guide for Professional 
Development,” CCSSO, 2001). Finally, we are 
conducting a three-year study to test the effects of using 
enacted curriculum data with school teams as a strategy 
for professional development to improve instruction in math 
and science (in collaboration with four urban districts, 40 
schools, TERC Regional Alliance, and support of NSF; for 
description, see http://www.ccsso.org/dec/dechome.htm) . 
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All of the CCSSO/WCER tools and services, and our list of publications, can be reviewed on our SEC 
website — http://www.ccsso.org/sec.html. 

The reports, instruments, and materials are available on compact disk-- SEC-CD - for locally-directed 
surveys and applications. 
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Topics Covered by Surveys 



The major concepts underlying the designs for the Surveys were drawn from state and national content 
standards, state initiatives in science and mathematics education, and prior research studies on 
classroom instructional practices and curriculum content. Each of the following major concepts can be 
addressed with the results of the Enacted Curriculum Surveys. With each concept, the data can be used 
to measure the amount of time spent teaching, differences among schools and classrooms, and variation 
in instruction by student characteristics. 

• Active Learning in Science 

• Problem Solving in Mathematics 

• Instructional Activities in classrooms, e.g., small groups, 

manipulatives, investigations 

• Mathematics and Science Content in Classrooms (see 

adjoining "content map" reporting format) 

• Multiple Assessment Strategies in Math and Science 

• Use of Education Technology and Equipment 

• Teacher Preparation in Subject 

• Quality of Professional Development 

• Influences of Policies and Standards on Practice 

• Alignment of Content Taught with State Assessments 
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Research and Development 



Survey Methodology 

The curriculum surveys and data analysis services being offered were developed through CCSSO’s 
State Collaborative on Assessment and Student Standards (SCASS) on Surveys of Enacted 
Curriculum, and through research and development conducted by Porter and Smithson at the 
Wisconsin Center. A multi-state team of educators, assessment specialists, and researchers worked 
with project leaders at CCSSO and WCER to write, review, and field test math and science surveys 
over a four-year period (1994-1998). The development work was supported by NSF. 

The surveys are based on state and national standards for content and teaching, as well as prior well- 
tested survey instruments, including TIMSS, NAEP, and National Survey of Science and Math 
Education (Horizon Research). Survey instruments were thoroughly field tested to ensure reliability and 
validity of the data. 

In Spring 1999, schools and teachers in 11 states participated in a Study of the Enacted Curriculum in 
Mathematics and Science classrooms. More than 600 teachers across the states completed self-report 
surveys that covered the subject content they taught and the instructional practices they used in their 
classes. The goals of the study were to: a) measure and report differences in instructional practices 
and curriculum content across the sample; b) determine the relationship of instruction to state policy 
initiatives and standards; and c) demonstrate the use of “surveys of enacted curriculum” to analyze 
classroom practices and report data for use by educators. 

Teachers participating in the 1999 Study reported on a full school-year of teaching in science or math, 
as has been done with previous administrations of the instruments. The best validation of the survey 
methodology, requiring teachers to recall instruction for a full school-year, comes from the Reform-Up- 
Close study (Porter, et al., 1993) where researchers collected and compared daily logs, independent 
observation, and teacher survey reports. This study found that data reported about curriculum content 
in teacher surveys covering a whole year were highly correlated with the data from daily logs of 
instructional content (Smithson and Porter, 1994). 

Findings from Research and Development 

Results from the 1999 data showed that educators can use the surveys and methods of analyzing data 
to address several kinds of problems and needs (Blank, et al., 2001). Specifically, the research studies 
showed that SEC instruments and reports do address educators needs for comparable, reliable data 
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and analyses of math and science instruction. The findings of our 1999 survey in over 200 schools had 
the following key results: 

• Demonstrated a high degree of variation in instructional time, practices and content of math and 
science from state to state, from school to school, and within schools 

• Curriculum taught in math and science differed according to amount of standards-based 
professional development in schools and level of implementation of state reform initiatives. 

• The method of surveying teachers using a “content matrix” based on standards, and then 
analyzing content of assessments using the same tool, showed strong potential for application in 
districts and states for measuring progress of standards-based math and science. 

• The comprehensive design of the surveys proved useful in analyzing the relationship of 
curriculum content and pedagogy to level of preparation of teachers, professional development, and 
school conditions for teaching. 

(See, Blank, Porter, & Smithson, New Tools for Analyzing Teaching, Curriculum and Standards in 

Mathematics & Science, CCSSO, 2001). 

To view this report and other SEC reports and materials, go to the CCSSO website: 
http://www.ccsso.org/sec.html. 




New Tools for Analyzing 
Teaching , Curriculum , and Standards 
in Mathematics & Science 



Results from the Surveys of Enacted Curriculum Project 
Final Rtfpcit 



CotrvietiKt hy\ 

Cow'tcit Of cnktf 5?a t* Scftoot Officers 
Wiscottsin C enter few Cduccfbo rteseaich 
E<ewn-Stett> OoteboraUve 




Ctaittil Oflwwr* 

« >iV MaitbUMMtIS « W, I*.' ifJRUfl -1 i i i 



ERIC 



6 





About Our Organizations 




The Council of Chief State School Officers (CCSSO) is a nationwide non-profit 
organization composed of the public officials who head departments of elementary 
and secondary education in the states, the District of Columbia, the Department of 
Defense Education Activity, and five extra-state jurisdictions. CCSSO seeks its 
members’ consensus on major education issues and expresses their view to civic 
and professional organizations, to federal agencies, Congress, and to the public. Through its structure 
of standing and special committees, the Council responds to a broad range of concerns about 
education and provides leadership on major education issues. 



' * f;NC' 



Because the Council represents each state’s chief education administrator, it has access to the 
educational and governmental establishment in each state and to the national influence that 
accompanies this unique position. CCSSO forms coalitions with many other education organizations 
and is able to provide leadership for a variety of policy concerns that affect elementary and secondary 
education. Thus, CCSSO members are able to act cooperatively on matters vital to the education of 
America’s young people. 

Council projects in the area of Education Indicators are aimed to improve the breadth, quality, and 
comparability of data on education, including state-by-state achievement data, instructional data, 
indicators of quality in areas such as mathematics and science, and performance assessment of 
teachers and students. In collaboration with state education agencies, the federal government, and 
national and international organizations, the Center contributes to the development of a set of useful 
and valid measures of educational quality geared, when appropriate, to education standards. 



Through the Surveys of Enacted Curriculum, CCSSO hopes to offer education systems a practical, 
efficient research tool for obtaining consistent data on math and science education based on teacher 
reports. The survey instruments and data analysis methods provide an objective approach for schools, 
districts, or states to analyze current classroom practices in relation to content standards. States have 
supported the development and testing of the tools, and federal grants, primarily from National Science 
Foundation, have supported research and development. The surveys were designed by CCSSO and a 
collaborative team, and items from several prior studies were included. 



CCSSO's program of developing, analyzing and reporting education indicators is directed by Rolf K. 
Blank, who is also leading the SEC effort team at CCSSO. Dr. Blank directed the 1999 study of state 
initiatives using the Surveys, and he has led the collaborative work with states and WCER in developing 
and testing the SEC model since 1994. 
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The Wisconsin Center for Education Research, originally established in 
1964 to house one of the university-based research and development centers 
created under the federal Cooperative Research Act, has maintained a longstanding mission of 
improving American education. WCER research spans the full scope of education, from the effects of 
infant child care to undergraduate curriculum reform. Much of the work focuses on the teaching, 
learning, and assessment of K-12 subjects: mathematics, science, social studies, and English. WCER 
is home to national centers for research on science, mathematics, engineering, and technology 
education; on school mathematics and science; on English learning and achievement; and on 
education policy. The Center combines and focuses the talents of scholars from different disciplines on 
the problems of teaching and learning, both in today's classrooms and in those of tomorrow. 

WCER’s annual extramural funding is over $15,000,000 for nearly 50 projects These projects range in 
size from $20,000 for Cognitive Studies of Interdisciplinary Collaboration in teacher education to 
$2,500,000 from the U.S. Department of Education for the National Center for Improving Student 
Learning and Achievement in mathematics and science. The WCER staff of over 200 include 46 faculty 
from academic departments across the University, 56 academic staff, and 95 graduate assistants. 

The SEC collaboration with CCSSO is led by Andrew Porter, director of WCER. Dr. Porter has 
conducted research and analysis on curriculum in math and science for two decades. In 1993, Porter 
developed a fine grained, three dimensional taxonomy of instructional content as part of the design for 
comparing the intended curricula of national, state, district and school policies with descriptions of 
classroom instruction and content. The study demonstrated that survey data were highly correlated 
with the results obtained from daily logs. Using a modified two-dimensional version of the content 
taxonomy developed by Porter, Gamoran et al. (1997) were able to link teacher surveys of instructional 
content to student achievement gains, yielding a correlation of .5 (a very significant finding). Now, the 
multidimensional model for describing instructional content has been used in SEC to create an 
alignment index for comparing instruction to assessments and standards. The content instrument 
developed by Porter and colleagues was also employed in a national evaluation of the Eisenhower 
Program on Designing Effective Professional Development (Porter, Garet, Birman, & Desimone,1999). 
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Applications — Needs of Educators Addressed by SEC 

For education decision-makers, the possible applications of SEC Tools and Services can be grouped 
into five categories. We find that these five categories address the main types of needs and problems 
in K-12 education for which these tools and services are appropriate and helpful: 



1. Monitor indicators of instruction and relationship to student achievement 

Data are collected using standard, self-report teacher surveys across classrooms and schools to allow 
comparison of science and math instructional practices and subject content. Standard instruments and 
survey items are used. A web-based survey and data collection is now being tested, but paper forms 
are available. Data can be analyzed by characteristics of schools (e.g., level, enrollment, reform 
initiative) by grade level within a school or across schools, or by teacher characteristics (e.g., high 
professional development in subject vs. low professional development). 

Student achievement data from local or state assessments can be reported along with the curriculum 
and instructional data from SEC tools. Repeat administration and use of the surveys allows schools 
and districts to analyze extent of change or improvement in instructional practices, and achievement, 
over time (e.g., annual survey or every two years). Monitoring of change in practices can be school or 
by teacher (if teachers are identified). 

Components of SEC for this application are: 

Planning (local leaders gain information about surveys and services, plan their use) 

Orientation of Users (introduce surveys to administrators and teachers, demonstrate data uses) 
Data Collection with Surveys (on-line or paper) 

Analysis and Report on Instruction and Achievement Results (analyze teacher responses on 
practices and subject content; guide for data interpretation; graphics for data display) 
Repeat data collection and reporting annually, or biennially. 



j Using Data on Enacted Curriculum - 1 
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2. Obtain survey instruments to collect enacted curriculum data 

Districts or states may want to obtain the SEC survey instruments and collect and analyze the data 
using the data analysis methods and formats developed by CCSSO and WCER. The surveys are 
available on a SEC-compact disk (CD) as well as the analysis and report format, and they will be 
available on the CCSSO website. 

Options : 

Purchase SEC- CD 

Access the CCSSO/WCER on-line SEC web-based surveys and report system 

Local districts or states would administer the surveys and conduct analysis of data using the models 
and data software provided on the CD or by gaining access to the on-line version of the tools. 

3. Align curriculum with standards or assessments 

Almost every state, and many districts, have developed content standards for student learning. The 
enacted curriculum surveys can provide a database for monitoring the degree to which classroom 
curriculum is moving toward the standards. States and districts are finding that one of the key benefits 
of the SEC tools is the capacity for analyzing the degree of intersection between the science and math 
subject curriculum content as taught in classrooms and content as tested on statewide or district-wide 
assessments. Procedures have been developed for content coding the assessments into the same 
content matrix used for the teacher survey providing teacher-reported data on what is taught. A 
mapping software from MSexcel is used to portray the degree of alignment between instructional 
content and assessed content. We recommend that districts organize a workshop for schools and 
teachers to analyze and interpret the results. 

Components: 

Planning 

Orientation of users 
Data collection 

Analysis and report, including Alignment analysis 

Interpretation of data through Workshop with school leadership teams and teachers 

4. Analyze differences in instruction among schools and classrooms, and assist school 
leadership teams to identify instructional improvement strategies 

CCSSO and WCER are currently conducting a project with four urban districts and 40 middle schools 
aimed toward improving instruction in math and science with use of the enacted curriculum data. The 
project is supported by a grant from National Science Foundation. 
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The model involves selection of school teams, data collection, reports for each school, workshops, and 
technical assistance. After two years, we will collect data again to analyze the extent of change in 
instructional practices and content. 

Components: 

Planning 

Orientation of users 
Data collection 

Analysis / Report- S-M Instruction, Alignment 
Interpretation of data / PD Workshop 

Technical assistance (visits to schools over 18 month period, follow-up workshops) 

Repeat data collection and analysis after 2 years 

5. Evaluate standards-based initiatives, e.g., professional development, to assess needs, 
determine effects, and assist planning r f 

Educators and decision-makers need reliable tools for evaluating the effects of initiatives aimed toward 
improving teaching practices and increasing student learning. The enacted curriculum surveys can 
provide a tool for identifying needs, determining effects on curriculum, or planning programs. In science 
and math, many states and districts have developed professional development models to improve 
teachers’ knowledge of subject content and their skills in teaching math and science. Methods of 
evaluating change in practices that provide valid, comparable results are difficult to design and 
implement. 

The SEC tools provide efficient survey-based methods of collecting data on current teaching practices, 
characteristics of teachers, and information on their professional development experiences, and then 
tracking change over time through repeat administrations of surveys. The effects of initiatives can be 
compared across schools and districts, and differing characteristics of teachers. 

Components: 

Planning 

Orientation of users 
Data collection 

Analysis / Report- S-M Instruction and Experience of teachers with initiative 
Interpretation of data in Workshop (survey reports provide key step in improving PD process) 
Repeat data collection and analysis at end of initiative, or after one year or two years 
experience. 
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Costs 

Estimates of the costs for obtaining SEC tools and services under each category are available from 
CCSSO. The cost estimates for each separate component are available as well. We encourage users 
to select the model or application that fits local or state needs and to arrange for the components 
needed in their system. 

For cost estimates and planning for your district or state, contact Rolf K. Blank, CCSSO Director of 
Education Indicators, rolfb@ccsso.org (202/336-7044), or John Smithson, WCER Project Manager, 
iohns@education.wisc.edu (608/263-6448). 

Further information on the SEC tools, services, reports, and available print and electronic materials are 
on the web: http://www.ccsso.org/sec.html. 
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Marketing Plan- -Providing information to potential users 



CCSSO, led by Rolf Blank, and WCER, led by Andrew Porter and John Smithson, along with other 
partners, will take several steps to market the “Surveys of Enacted Curriculum: Tools and Services” to 
educators and decision-makers with education systems. 

The primary targets of the SEC Marketing Plan are decision-makers for state and local district 
education systems. Primary users of the SEC are administrators, curriculum specialists, and program 
leaders at the school, district, and state levels. Because use of the SEC requires commitment of staff 
time and funding resources, marketing should provider information to users but ultimately focus on 
needs of decision-makers. 

The goal of the marketing plan is to provide information on SEC tools and services to potential users in 
order that they can: a) learn how SEC data will benefit their teachers and students and improve their 
work as educators, b) determine how they gain access to SEC and identify what kinds of data and 
services they will receive; and c) encourage a commitment toward the SEC by system decision- 
makers. 

Marketing Activities 

1 . Provide electronic access to information about the Surveys, and link to web survey- 

a) Potential users go through the CCSSO website to www.ccsso.org/SEC.html - where they 

can view descriptions of the SEC Tools and description of Services, where they can purchase 
SEC-CD, and where they can connect to online web-based Surveys (at WCER website — 
http://www.wcer.wisc.edu/ ) . 

b) Users gain access to SEC through a link from a partner website- e.g. WCER, NSTA, other 
organizations that partner by listing and briefly describing SEC services 

c) Send emails directly to education decision-makers or key contacts (e.g., NSTA listserve) 
which describe SEC and provide the web address at CCSSO and web survey link 

d) Disseminate information through listserves, e.g., AERA Div H (sent by RB), ASSM, CSSS, 

2. Make presentations at professional conferences, meetings (Blank, Porter, Smithson, others) 

a) submit proposals to present findings from existing studies (AERA, NCME, LSAC, MIS, 
science and math conferences) 

b) request/arrange for invited demonstration of tools and potential uses (e.g., at prof, 
organizations, e.g., NASSP; an SEA, NSF brown bag, school district) 

13 
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3. Publish articles in education journals, magazines, esp. those serving education decision-makers 

a) Focus writing on how SEC data can be used-- emphasis on aligning instruction with 
standards/assessments- e.g., Administrator, School Principal, ASCD Ed. Leadership, 
Eisenhower Clearinghouse magazine/website 

b) Articles in professional newsletters (e.g., mailing of 11/30 to education organizations) 

4. Build on contacts/leaders from DEC project, SEC collaborative, NSF 

a) Establish connections with other sites receiving NSF support, e.g., USP project directors, 
LSC districts, states proposing math/science partnerships 

b) Expand to other schools in the 4 DEC districts 

c) Make presentation at NSF- sponsored meetings of supported sites 

d) Use SEC collaborative states’ representatives to provide information to other potential 
users and sites 

5. Provide data collection support and assistance to 10-sites (2001-03) thru NSF support for 
testing of web-based Surveys 

a) Pilot testing with selected schools-up to 10 sites, Spring 2002; provide report of pilot 
results to sites, Fall 2002 

b) Test web surveys and reporting with full sample of schools in each site, 2002-03 SY 
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Operational Plan-- Assisting education systems to use SEC tools and services 

Educators or administrators may have obtained information about the SEC tools and services in a 
variety of ways. Requests from education systems to obtain SEC tools and services can be made 
through one of three routes, as of February 2002: 

A) Direct contact inquiry for information or assistance to one of the staff at CCSSO or WCER by 
phone, email, or letter. In response, we would: 

- Discuss the requested services, and needs to be met, with users by phone or email 

- Email sample materials and descriptions of services 

- Negotiate arrangements to use surveys and services, determine method of support and 
payment, set schedule for providing services, send contract to user (see attached spreadsheet analysis 
of costs) 

- Project manager (from CCSSO or WCER) will be assigned to each participating site 

B) Purchase SEC- CD (containing all survey instruments, reports, papers, and professional 
development materials); or purchase hard-copy report or other publications from CCSSO. 

- Contact CCSSO publications office by phone or through CCSSO website 

C) Access On-line, Web-based SEC system [access and request forms to be developed] 

Steps in conducting data collection and receiving reports: 

1. School users review surveys and report options; make decisions on grades, subject, 
sections to be used; and complete request form on-line (sent to central server/ data manager 
for processing) 

2. Manager confirms requested services and arrangements, and transmits contract amount 

3. Method of payment (e.g., purchase order or credit card) returned by user 

4. Access code provided by manager to user for web surveys 

5. Procedures and schedule relayed to user 

6. Names and credentials of staff/consultants, and contact information are provided for 
partners that will provide assistance with local analysis/interpretation, technical assistance or 
workshop services 

7. Teachers complete surveys on-line within designated time period 

8. On-line feedback to users with comparison of individual responses with total sample 

9. Data analysis based on pre-designed format conducted centrally, if requested and paid 

10. Analytical report delivered by email attachment- based on school/district responses 
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Key Contacts 

Rolf K. Blank (SEC Project Director) 
Director of Education Indicators 
Council of Chief State School Officers 
One Massachusetts Ave. NW, Suite 700 
Washington, DC 20001-1431 
202/336-7044; F 202/789-1792 
rolfb@ccsso.org 

Cynthia Pewett 

Senior Project Associate 

Council of Chief State School Officers 

One Massachusetts Ave. NW, Suite 700 

Washington, DC 20001-1431 

202/336-7045; F 202/789-1792 

cvnthiap@ccsso.org 



Andrew Porter 

Director, Wisconsin Center for Education Research 
1025 W. Johnson St. 

Madison, Wl 53706 
608/263-4200; F 608/263-6448 
andyp@education.wisc.edu 

John Smithson 
Project Manager 

Wisconsin Center for Education Research 
1025 W. Johnson St. 

Madison, Wl 53706 
608/263-44354; F 608/262-7428 
johns@education.wisc.edu 
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